Pupil tracking device

ABSTRACT

There is provided a pupil tracking device including an active light source, an image sensor and a processing unit. The active light source emits light toward an eyeball alternatively in a first brightness value and a second brightness value. The image sensor captures a first brightness image corresponding to the first brightness value and a second brightness image corresponding to the second brightness value. The processing unit identifies a brightest region at corresponding positions of the first brightness image and the second brightness image as an active light image.

CROSS REFERENCE TO RELATED APPLICATION

This application is a continuation application of U.S. application Ser.No. 16/213,043, filed on Dec. 7, 2018, which is a continuationapplication of U.S. application Ser. No. 15/253,503, filed on Aug. 31,2016, which is a continuation application of U.S. application Ser. No.14/054,724, filed on Oct. 15, 2013 and claims the priority benefit ofTaiwan Patent Application Serial Number 101139163, filed on Oct. 23,2012, the full disclosures of which are incorporated herein byreference.

BACKGROUND 1. Field of the Disclosure

This disclosure generally relates to an interactive system and, moreparticularly, to a pupil tracking device that includes at least oneactive light source to be served as a positioning reference point.

2. Description of the Related Art

As the interactive control mechanism can provide more intuitiveoperation to users, it has been widely applied to various multimediasystems, especially to an image display system having a display screen.

A remote controller capable of capturing images is generally served as ahuman-machine interface and the remote controller can be manufactured asvarious properties, such as a bat, a racket, a club and so on. There isanother kind of interactive human-machine interface that does not needany hand-held device. For example, a pupil tracking device can performthe interactive operation according to the variation of a line of sightof the user.

The conventional pupil tracking device has the problem that it is easilyaffected by ambient light sources. For example, FIG. 1 shows a schematicdiagram of an eye image 9, which contains a pupil image Ip and anambient light source image Io. When the pupil image Ip overlaps theambient light source image Io, it may not be able to position the pupilimage Ip correctly such that the pupil tracking may not be performedcorrectly.

Accordingly, the present disclosure further provides a pupil trackingdevice that may eliminate the effect from ambient light sources therebyimproving the accuracy of pupil tracking.

SUMMARY

The present disclosure provides a pupil tracking device that includes atleast one active light source emitting light toward an eyeball to form apositioning reference point.

The present disclosure further provides a pupil tracking device suitableto be applied to a head accessory.

The present disclosure provides a pupil tracking device including alight source, an image sensor and a processing unit. The light source isconfigured to emit light toward an eyeball with different brightnessvalues at different times to form a first reflection spot and a secondreflection spot on the eyeball, respectively. The image sensor isconfigured to capture a first image containing the first reflection spotand a second image containing the second reflection spot, wherein theimage sensor is further configured to capture a third image when thelight source is being turned off. The processing unit is electricallycoupled to the image sensor and the light source, and configured tocalculate a first differential image between the first image and thethird image, and calculate a second differential image between thesecond image and the third image; calculate a differential image betweenthe first differential image and the second differential image, andidentify a darkest region in the differential image as an image of thelight source projected on the eyeball.

The present disclosure further provides a pupil tracking deviceincluding a light source, an image sensor and a processing unit. Thelight source is configured to emit light toward an eyeball withdifferent brightness values at different times to form a firstreflection spot and a second reflection spot on the eyeball,respectively. The image sensor is configured to capture a first imagecontaining the first reflection spot and a second image containing thesecond reflection spot, wherein the image sensor is further configuredto capture a third image when the light source is being turned off. Theprocessing unit is electrically coupled to the image sensor and thelight source, and configured to calculate a first differential imagebetween the first image and the third image, and calculate a seconddifferential image between the second image and the third image;calculate a differential image by subtracting the second differentialimage from the first differential image, and identify one of two darkestregions in the differential image having a smaller area andcorresponding to a position of a brightest region in the first or seconddifferential image as an image of the light source projected on theeyeball.

The present disclosure further provides a pupil tracking deviceincluding a light source, an image sensor and a processing unit. Thelight source is configured to emit light toward an eyeball with fixedbrightness value at different times to form a first reflection spot anda second reflection spot on the eyeball, respectively. The image sensoris configured to capture, using a first exposure time, a first imagecontaining the first reflection spot, and capture, using a secondexposure time different from the first exposure time, a second imagecontaining the second reflection spot, wherein the image sensor isfurther configured to capture a third image when the light source isbeing turned off. The processing unit is electrically coupled to theimage sensor and the light source, and configured to calculate a firstdifferential image between the first image and the third image, andcalculate a second differential image between the second image and thethird image; calculate a differential image between the firstdifferential image and the second differential image, and identify adarkest region in the differential image as an image of the light sourceprojected on the eyeball.

In one aspect, the processing unit is further configured to identify apupil position according to the first brightness image or the secondbrightness image, wherein when the first brightness value is higher thanthe second brightness value, the processing unit identifies a pluralityof pixels around a lowest gray level in the first brightness image andhaving gray levels within a gray level range as a pupil region, andcalculates a gravity center or a center of the pupil region to be servedas the pupil position.

In one aspect, the pupil tracking device further includes a memory unitconfigured to save pupil coordinate information associated with theactive light image and the pupil position. The processing unit maycalculate a current pupil coordinate according to the pupil coordinateinformation.

In one aspect, the first brightness image and the second brightnessimage may be implemented by changing the exposure time and/or the gainof the image sensor without changing the brightness value of the activelight source.

In one aspect, the processing unit further identifies a first pupilregion according to the first image frame and identifies a second pupilregion according to the second image frame, and defines a joined regionof the first pupil region and the second pupil region as an outputtedpupil region.

In one aspect, the pupil tracking device includes four active lightsources, and the processing unit identifies four active light images inthe differential image and respectively calculates a positioningcoordinate associated with the four active light images. The processingunit may further calculate an eyeball center according to the fourpositioning coordinates.

In one aspect, the processing unit is further configured to identify anoutputted pupil region according to the first image frame and the secondimage frame, and define a direction of connecting line of the eyeballcenter and the outputted pupil region as a direction of line of sight.

In the pupil tracking device according to the embodiment of the presentdisclosure, the active light image is referred to the reflection imageof the active light source on the cornea.

In the pupil tracking device according to the embodiment of the presentdisclosure, the active light image is served as a positioning referencepoint in pupil tracking. Therefore, the pupil tracking device ispreferably disposed on a head accessory so as to fix the position of theactive light image in the eyeball.

In the pupil tracking device according to the embodiment of the presentdisclosure, the accuracy of pupil tracking may be effectively improvedby accurately identify the position of the active light image to beserved as the positioning reference point and by calculating thedifferential image to eliminate the effect from ambient light sources.

BRIEF DESCRIPTION OF THE DRAWINGS

Other objects, advantages, and novel features of the present disclosurewill become more apparent from the following detailed description whentaken in conjunction with the accompanying drawings.

FIG. 1 shows a schematic diagram of the eye image.

FIG. 2A shows a schematic block diagram of the pupil tracking deviceaccording to an embodiment of the present disclosure.

FIG. 2B shows a schematic diagram of the pupil coordinate information inthe pupil tracking device according to the embodiment of the presentdisclosure

FIG. 3A shows an operational schematic diagram of the pupil trackingdevice according to an embodiment of the present disclosure.

FIG. 3B shows a schematic diagram of the image capturing and thelighting of the light source of the pupil tracking device according toan embodiment of the present disclosure.

FIG. 3C shows a schematic diagram of identifying the active light imagein the pupil tracking device according to an embodiment of the presentdisclosure.

FIG. 3D shows a schematic diagram of the image capturing and thelighting of the light source of the pupil tracking device according toanother embodiment of the present disclosure.

FIG. 4 shows a schematic diagram of identifying the pupil position inthe pupil tracking device according to the embodiment of the presentdisclosure.

FIG. 5A shows an operational schematic diagram of the pupil trackingdevice according to another embodiment of the present disclosure.

FIG. 5B shows a schematic diagram of the image capturing and thelighting of the light source of the pupil tracking device according toanother embodiment of the present disclosure.

FIG. 5C shows a schematic diagram of identifying the active light imagein the pupil tracking device according to another embodiment of thepresent disclosure.

DETAILED DESCRIPTION OF THE EMBODIMENT

It should be noted that, wherever possible, the same reference numberswill be used throughout the drawings to refer to the same or like parts.

Referring to FIG. 2A, it shows a schematic block diagram of the pupiltracking device according to an embodiment of the present disclosure,which includes at least one active light source (e.g. four active lightsources 111 to 114 shown herein), an image sensor 12, a processing unit13 and a memory unit 14. The pupil tracking device 1 of this embodimentuses the active light source to project at least one active light imageon an eyeball 90 to be served as a positioning reference point, anddetermines a pupil coordinate according to the relationship between apupil position and the positioning reference point (i.e. the activelight image), wherein the active light image is referred to a reflectionimage of the active light source on the cornea. The pupil trackingdevice 1 may wired or wirelessly transmit the pupil coordinate to animage display device 3 (FIG. 2B) to accordingly control the imagedisplay device 3 to perform a corresponding function. It should bementioned that the purpose of using four active light sources is tocalculate an eyeball center, but if it is to perform the pupil trackingonly, any number of the active light sources may be used.

Generally speaking, when the eyeball 90 looks downward, the upper eyelidcovers a part of the eyeball 90. Therefore, if the pupil tracking device1 is disposed on a head accessory 2, the disposed position of the imagesensor 12 is preferably lower than the eyeball 90 to avoid the pupilcannot be detected when the eyeball 90 looks downward (i.e. the pupilmoving downward).

The active light sources 111 to 114 may be infrared light sources, e.g.infrared light emitting diodes, to avoid affecting the vision duringlighting. The active light sources 111 to 114 are configured to emitlight toward the eyeball 90 so as to illuminate the eyeball 90 and formreflection light spots on the eyeball 90 (i.e. the active light imageson the eyeball 90). It should be mentioned that each active light sourcemay include only one light emission element or arranged by a pluralityof light emission elements.

The image sensor 12 may be the sensor configured to sense optical energysuch as a CCD image sensor and a COMS image sensor, and captures imageframes of the eyeball 90 at a frame rate corresponding to the lightingof the active light sources 111 to 114. The image sensor 12 may captureimage frames with a fixed or variable exposure time and gain. It shouldbe mentioned that the frame rate of the image sensor 12 may not be equalto the lighting frequency of the active light sources 111 to 114.

The processing unit 13 is configured to control the lighting of theactive light sources 111 to 114 and the image capturing of the imagesensor 12, and to identify the pupil position, pupil coordinate, activelight image, eyeball center and direction of line of sight according tothe image frames captured by the image sensor 12 (described later). Inother embodiments, the processing unit 13 may also be formed by aplurality elements without any limitation, e.g. an independent lightcontrol unit for controlling the active light sources 111 to 114 and anindependent digital signal processor (DSP) for performing the imagepost-processing.

The memory unit 14 is configured to save pupil coordinate informationassociated with the active light image and the pupil position. In thismanner, when the processing unit 13 obtains a current relationshipbetween the current pupil position and the active light image, a currentpupil coordinate may be obtained according to the previously saved pupilcoordinate information. For example referring to FIG. 2B, the pupiltracking device 1 according to the embodiment of the present disclosuremay include a setup procedure that may be automatically entered beforeeach operation or entered according to an instruction. In the setupprocedure, a user may sequentially stare at predetermined referencepoints, e.g. four corners C₁ to C₄ of the image display device 3, butnot limited to. Meanwhile the image sensor 12 sequentially captures foureye images 9 ₁ to 9 ₄, and each of the eye images 9 ₁ to 9 ₄respectively contains a pupil image I_(C1) to I_(C4) and an active lightimage I₁₁₁, wherein the active light image I₁₁₁ locates at the same (orcorresponding) position in the four eye images 9 ₁ to 9 ₄. For example,when the user stares at the reference point C₁, the image sensor 12captures the eye image 9 ₁, wherein a vector from the pupil image I_(C1)to the active light image I₁₁₁ is assumed to be {right arrow over (d₁)};similarly, the processing unit 13 may respectively calculate vectors{right arrow over (d₂)} to {right arrow over (d₄)} in the eye images 9 ₂to 9 ₄. The pupil tracking device 1 then defines an operating spaceaccording to the vectors {right arrow over (d₁)} to {right arrow over(d₄)} to be previously saved in the memory unit 14. In this manner, inactual operation the processing unit 13 may calculate a current pupilcoordinate according to the operating space. It should be mentioned thatdirections of the vectors {right arrow over (d₁)} to {right arrow over(d₄)} in FIG. 2B may be reversed.

Referring to FIG. 3A, it shows an operational schematic diagram of thepupil tracking device 1 according an embodiment of the presentdisclosure. The pupil tracking device 1 includes an active light source111, an image sensor 12, a processing unit 13 and a memory unit 14. Asmentioned above the memory unit 14 is configured to save pupilcoordinate information associated with the active light image and thepupil position, and thus details thereof are not repeated herein.

Referring to FIGS. 3A-3D, FIG. 3B shows a schematic diagram of the imagecapturing of the image sensor 12 and the lighting of the active lightsource 111 in the pupil tracking device of FIG. 3A; FIG. 3C shows aschematic diagram of identifying the active light image I₁₁₁ by theprocessing unit 13 in the pupil tracking device of FIG. 3A; and FIG. 3Dshows another schematic diagram of the image capturing of the imagesensor 12 and the lighting of the active light source 111 in the pupiltracking device of FIG. 3A.

The active light source 111 emits light toward an eyeball 90alternatively in a first brightness value and a second brightness value.The image sensor 12 captures a first brightness image f_(b1)corresponding to the first brightness value and a second brightnessimage f_(b2) corresponding to the second brightness value. For examplein FIG. 3C, the first brightness image f_(b1) contains a brightestregion A₁₁₁ (i.e. the active light image), a pupil image 91, an irisimage 92 and a cornea area 93. The second brightness image f_(b2)contains a brightest region A₁₁₁′ (i.e. the active light image), a pupilimage 91′, an iris image 92′ and a cornea area 93′. As the pupiltracking device 1 according to the embodiment of the present disclosureis preferably disposed on a head accessory 2 to allow the active lightsource 111 to keep a fixed relative position with respect to the eyeball90, the brightest regions A₁₁₁ and A₁₁₁′ are at corresponding (oridentical) positions in different brightness images. The active lightimage is the direct reflection point of the active light source 111 onthe cornea, and thus the active light image is the brightest region inthe brightness image. Therefore, the processing unit 13 may identify abrightest region at the corresponding positions (i.e. A₁₁₁ and A₁₁₁′) inthe first brightness image f_(b1) and the second brightness image f_(b2)as an active light image I₁₁₁, wherein said brightest region is referredto the region having the highest gray level. As the space relationshipbetween the eyeball 90 and ambient light sources is not fixed, theambient light source image does not locate at the same position indifferent brightness images. Accordingly, although there may be otherambient light source images contained in the brightness image capturedby the image sensor 12, the effect from ambient light sources may beeliminated by searching the brightest region at corresponding positions.In other embodiments, the image sensor 12 may further capture an image(e.g. a third brightness image) f_(b3) when the active light source 111is turned off as shown in FIG. 3D, and then subtracts the thirdbrightness image f_(b3) from the first brightness image f_(b1) and thesecond brightness image f_(b2) respectively to obtain f_(b13) andf_(b23) so as to eliminate the interference from ambient light andstabilize the detection of the active light image of the brightestregion.

Furthermore, in addition to changing the brightness value of the activelight source 11 (e.g. changing the drive current thereof), the sameobject may be achieved by fixing the brightness value of the activelight source 111 but changing the sampling parameter (e.g. including theexposure time and gain) of the image sensor 12. That is, in anotherembodiment the first brightness value is equal to the second brightnessvalue, and the image sensor 12 captures the first brightness imagef_(b1) with a first exposure time and a first gain and captures thesecond brightness image f_(b2) with a second exposure time and a secondgain, wherein the first exposure time is not equal to the secondexposure time and/or the first gain is not equal to the second gain.

In addition, in this embodiment the processing unit 13 may furtheridentify a pupil position according to the first brightness image f_(b1)or the second brightness image f_(b2), wherein the processing unit 13preferably identifies the pupil position according to the brightnessimage having higher brightness so as to clearly identify the pupil image91. For example referring to FIG. 4, the processing unit 13 isconfigured to calculate a lowest gray level P₁ in the first brightnessimage f_(b1) (e.g. assuming the first brightness value is higher thanthe second brightness value) and to identify a plurality of pixelsaround the lowest gray level P₁ and having gray levels within a graylevel range Rg as a pupil region PA. When the active light source 11 isturned on, the pupil image 91 has the lowest brightness, the cornea area93 has the highest brightness and the iris image 92 has the brightnessbetween the pupil image 91 and the cornea area 93, and thus the lowestgray level P₁ appears in the pupil image 91. Therefore, the pixel regionneighboring to the lowest gray level P₁ may be defined as the pupilregion PA. The pixel region neighboring to the lowest gray level P₁ andadjacent to each other may be defined as a same object by using theimage grouping, for example referred to U.S. Patent Pub. No.2011/0176733, entitled “Image recognition method” and assigned to thesame assignee of the present application. In addition, the gray levelrange Rg may be set according to the operation environment of the pupiltracking device 1, e.g. different gray level ranges Rg may be set foroutdoor and indoor use. Furthermore, to avoid noise interference, theprocessing unit 13 may further identify whether the pupil region PA isbelong to the ambient light source image according the size and shapethereof, e.g. when the size is too small and the shape is not a circle,it may be the ambient light source image and is removed.

Next, the processing unit 13 may calculate a gravity center or a centerof the pupil region PA to be served as a pupil position P₂, and output apupil coordinate (x,y) according to a relative position between thepupil position P₂ and the active light image I₁₁₁ and referencing thepupil coordinate information saved in the memory unit 14. The processingunit 13 may relatively control a cursor motion on an image displaydevice 3 according to the pupil coordinate (x,y). It is appreciated thatthe pupil position P₂ may not be equal to the position of the lowestgray level P₁.

In a word, in this embodiment the processing unit 13 identifies theactive light image I₁₁₁ according to two image frames but identifies thepupil position P₂ with only one image frame (e.g. the brighter image) soas to eliminate the interference from ambient light sources and theactive light image is served as a positioning reference point in pupiltracking.

In another embodiment, in order to further eliminate the influence fromambient light sources, the processing unit 13 may further calculate adifferential image (f_(b1)−f_(b2)) between the first brightness imagef_(b1) and the second brightness image f_(b2), and compare a darkestregion in the differential image (f_(b1)−f_(b2)) with the firstbrightness image f_(b1) or the second brightness image f_(b2) so as toidentify an active light image I₁₁₁″ (as shown in FIG. 3C), wherein saiddarkest region is referred to the region having the lowest gray level,e.g. about 0 gray level or a relative lowest with respect to other pixelgray levels.

More specifically speaking, as the active light image I₁₁₁ (i.e. thebrightest region A₁₁₁ and A₁₁₁′) has the highest brightness (e.g. thehighest gray level of the used gray level scale) in the first brightnessimage f_(b1) and the second brightness image f_(b2), the active lightimage I₁₁₁″ in the differential image (f_(b1)−f_(b2)) has the lowestbrightness. For example in one embodiment, it is assumed that the activelight image I₁₁₁ is 255 gray level, the pupil image 91 is 0 gray level,the iris image 92 is 128 gray level and the cornea area 93 is 228 graylevel in the first brightness image f_(b1); and the active light imageI₁₁₁′ is 255 gray level, the pupil image 91′ is 0 gray level, the irisimage 92′ is 64 gray level and the cornea area 93′ is 100 gray level inthe second brightness image f_(b2). Therefore, it is obtained that theactive light image I₁₁₁″ is 0 gray level, the pupil image 91″ is 0 graylevel, the iris image 92″ is 64 gray level and the cornea area 93″ is128 gray level in the differential image (f_(b1)−f_(b2)). As the pupilimages 91 and 91′ are both 0 gray level, the pupil image 91″ in thedifferential image (f_(b1)−f_(b2)) also has the lowest brightness. Now,the position of the active light image I₁₁₁″ (as shown in FIG. 3C) maybe identified by comparing the differential image (f_(b1)−f_(b2)) withthe first brightness image f_(b1) or the second brightness image f_(b2).Since the pupil images 91 and 91′ have the lowest brightness in thefirst brightness image f_(b1) or the second brightness image f_(b2), themethod of comparison may be performed by identifying the image objectwhose gray level changes from relative brightest to darkest whencomparing the first brightness image f_(b1) with the differential image(f_(b1)−f_(b2)) so as to accordingly determine the position of theactive light image I₁₁₁″ (as shown in FIG. 3C), or identifying the imageobject whose gray level changes from relative brightest to darkest whencomparing the second brightness image f_(b2) with the differential image(f_(b1)−f_(b2)) so as to accordingly determine the position of theactive light image I₁₁₁″ (as shown in FIG. 3C). In another embodiment,it is able to identify the object image in the differential image(f_(b1)−f_(b2)) having the lowest brightness and a smaller area as theactive light image and identify the object image having the lowestbrightness and a larger area as the pupil image.

Furthermore, in this embodiment in addition to that the calculation ofthe active light image I₁₁₁ is performed by using the differential image(f_(b1)−f_(b2)), the method of identifying the pupil position,constructing the pupil coordinate information and implementing the firstbrightness image f_(b1) and the second brightness image f_(b2) issimilar to the previous embodiment, and thus details thereof are notrepeated herein.

Referring to FIG. 5A, it shows an operational schematic diagram of thepupil tracking device 1 according to another embodiment of the presentdisclosure. The pupil tracking device 1 includes four active lightsources 111 to 114, an image sensor 12, a processing unit 13 and amemory unit 14. As mentioned above the memory unit 14 is configured tosave pupil coordinate information associated with the active light imageand the pupil position, and thus details thereof are not repeatedherein.

Referring to FIGS. 5A-5C, FIG. 5B shows a schematic diagram of the imagecapturing of the image sensor 12 and the lighting of the active lightsources 111 to 114 in the pupil tracking device of FIG. 5A; and FIG. 5Cshows a schematic diagram of identifying the active light image by theprocessing unit 13 in the pupil tracking device of FIG. 5A. As mentionedabove, the purpose of using four active light sources is to calculatethe eyeball center, and if it is to perform the pupil tracking only,using at least two active light sources may achieve the object of thepresent embodiment.

The active light sources 111 to 114 alternatively emit light toward aneyeball 90 so as to form reflection light spots LS₁₁₁ to LS₁₁₄ on theeyeball 90. The image sensor 112 captures a first image frame f1 and asecond image frame f2 corresponding to the lighting of differentcombinations of the active light sources 111 to 114. The processing unit13 is configured to calculate a differential image (f1−f2) between thefirst image frame f1 and the second image frame f2, and identify adarkest region (e.g. I₁₁₃′ and I₁₁₄′) and a brightest region (e.g. I₁₁₁′and I₁₁₂′) in the differential image (f1−f2) as the active light image.

For example in FIG. 5C, when the active light sources 111 and 112 areturned on (the active light sources 113 and 114 turned off now), theimage sensor 12 captures a first image frame f1, which contains a pupilimage 91, an iris image 92, a cornea area 93 and reference active lightimages I₁₁₁ and I₁₁₂; and when the active light sources 113 and 114 areturned on (the active light sources 111 and 112 turned off now), theimage sensor 12 captures a second image frame f2, which contains a pupilimage 91′, an iris image 92′, a cornea area 93′ and reference activelight images I₁₁₃ and I₁₁₄, wherein the light source brightnessassociated with the first image frame f1 and the light source brightnessassociated with the second image frame f2 may or may not identicalwithout any limitation. In addition in other embodiments, at least oneof the active light sources 111 to 114 (e.g. one or three active lightsources) are turned on corresponding to the capturing of the first imageframe f1 and the rest active light sources (e.g. three or one activelight source) are turned on corresponding to the capturing of the secondimage frame f2; i.e. it is not limited to turn on two active lightsources at the same time. When the second image frame f2 is subtractedfrom the first image frame f1, the reference active light images I₁₁₁and I₁₁₂ become the brightest regions to have the highest gray level,and the reference active light images I₁₁₃ and I₁₁₄ become the darkestregions to have the lowest gray level in the differential image (f1−f2).Accordingly, the processing unit 13 may identify four active lightimages I₁₁₁′ to I₁₁₄′ in the differential image (f1−f2). In oneembodiment, it is assumed that in the first image frame f1 the activelight images I₁₁₁ and I₁₁₂ have about +255 gray level, the pupil image91 has about 0 gray level, the iris image 92 has about 64 gray level andthe cornea area 93 has about 128 gray level; and in the second imageframe f2 the active light images I₁₁₃ and I₁₁₄ have about +255 graylevel, the pupil image 91′ has about 0 gray level, the iris image 92′has about 64 gray level and the cornea area 93′ has about 128 graylevel. Accordingly, it is obtained that in the differential image(f1−f2) the active light images I₁₁₁′ and I₁₁₂ are about +127 graylevel, the active light images I₁₁₃′ and I₁₁₄′ are about −127 gray leveland other parts are about 0 gray level. In another embodiment, it isable to adjust all gray levels to allow every gray level to be largerthan or equal to zero. For example, the active light images I₁₁₁′ andI₁₁₂ may be adjusted to about +254 gray level, the active light imagesI₁₁₃′ and I₁₁₄′ may be adjusted to about 0 gray level and other partsare adjusted to about +127 gray level. It should be mentioned that theabove gray levels are only exemplary.

As four active light sources 111 to 114 are used in this embodiment,after identifying the four active light images I₁₁₁′ to I₁₁₄′ in thedifferential image (f1−f2), the processing unit 13 may respectivelycalculate a positioning coordinate associated with the four active lightimages I₁₁₁′ to I₁₁₄′, and each of the positioning coordinates may be athree-dimensional coordinate. The processing unit 13 may calculate aneyeball center of the eyeball 90 according to the four positioningcoordinates, e.g. calculating by the sphere equation(x−x₀)²+(y−y₀)₂+(z−z₀)²=r². In this manner, the processing unit 13 mayfurther define a direction of connection line of the eyeball center andan outputted pupil region as a direction of line of sight.

In this embodiment, the outputted pupil region may be determined asfollowing. First, the processing unit 13 identifies a first pupil regionaccording to the first image frame f1 and a second pupil regionaccording to the second image frame f2. The method of identifying thepupil region is similar to FIG. 4; i.e. the processing unit 13identifies a plurality of pixels around a lowest gray level (e.g. P₁) inthe first image frame f1 and having gray levels within a gray levelrange Rg as the first pupil region (e.g. PA); and the processing unit 13identifies a plurality of pixels around a lowest gray level (e.g. P₁) inthe second image frame f2 and having gray levels within the gray levelrange Rg as the second pupil region (e.g. PA). After the first pupilregion and the second pupil region are identified, the processing unit13 calculates a gravity center or a center of the first pupil region tobe served as a first pupil position (e.g. P₂) and calculates a gravitycenter or a center of the second pupil region to be served as a secondpupil position (e.g. P₂).

As two image frames may be used to obtain the pupil image in thisembodiment, the processing unit 13 may define a joined region of thefirst pupil region and the second pupil region as an outputted pupilregion. Similarly, the processing unit 13 may determine an outputtedpupil coordinate according to the outputted pupil region and the pupilcoordinate information previously saved in the memory unit 14, and thefour active light images are served as positioning reference pointsherein. In other embodiments, the processing unit 13 may select one ofthe first pupil region and the second pupil region as the outputtedpupil region.

In addition, the pupil tracking device 1 according to the embodiment ofthe present disclosure may cooperate with a display unit for displayingimages, and the display unit may be integrated on the head accessory 2,e.g. glasses or a goggle.

The pupil tracking device 1 according to the embodiment of the presentdisclosure may further have the blinking detection function. Forexample, the processing unit 13 may record time intervals in which thepupil is detected and not detected respectively thereby identifying theblinking action.

The pupil tracking device 1 according to the embodiment of the presentdisclosure may further have sleep detection and distraction detectionfunctions. For example, when the pupil tracking device 1 is applied tothe vehicle devices, it is able to detect whether the driver is sleepyor watches ahead and give a warning in suitable time. The sleepdetection may be implemented by detecting the time ratio of eye closingand eye opening, and the distraction detection may be implemented bydetecting the direction of line of sight of the driver.

The pupil tracking device 1 according to the embodiment of the presentdisclosure may further have blinking frequency detection and dry eyedetection functions. More specifically speaking, the processing unit 13may derive the possibility and extend of dry eye according to thedetected blinking frequency and notify the user to blink his or hereyes.

The pupil tracking device 1 according to the embodiment of the presentdisclosure may further have the gesture recognition function. Thegesture recognition mentioned herein may be, for example, detecting thepupil moving toward a predetermined direction for predetermined timesand comparing the detected results with predetermined gestures so as toachieve the operation of performing specific function(s). The gesturerecognition is similar to the hand gesture recognition using other partsof the body, e.g. the hand or finger(s).

The pupil tracking device 1 according to the embodiment of the presentdisclosure may further have the power saving function. For example, apower saving mode may be entered when the pupil is not detected or theimage variation between image frames is too small for a predeterminedtime interval.

It is appreciated that the ratio of the active light image and the pupilimage in the drawings herein are only exemplary.

It should be mentioned that the pupil tracking device 1 according to theembodiment of the present disclosure may be directly manufactured as ahead mounted pupil tracking device or may be fixed, using a securingmember, on a head accessory, e.g. the glasses, goggle or hat that isworn, covered or mounted on the user's head. Preferably the active lightsource does not have relative movement with respect to the human eye,and the active light image is served as the positioning reference pointin pupil tracking.

As mentioned above, as the conventional pupil tracking device is notable to eliminate the interference from ambient light sources,misdetection may be caused during operation. Therefore, the presentdisclosure further provides a pupil tracking device (FIGS. 2A, 3A and5B) that includes an active light source having a fixed relativeposition with respect to the eyeball to be served as a positioningreference point so as to eliminate the interference from ambient lightsources thereby improving the detection accuracy.

Although the disclosure has been explained in relation to its preferredembodiment, it is not used to limit the disclosure. It is to beunderstood that many other possible modifications and variations can bemade by those skilled in the art without departing from the spirit andscope of the disclosure as hereinafter claimed.

What is claimed is:
 1. A pupil tracking device, comprising: a lightsource configured to emit light toward an eyeball with differentbrightness values at different times to form a first reflection spot anda second reflection spot on the eyeball, respectively; an image sensorconfigured to capture a first image containing the first reflection spotand a second image containing the second reflection spot, wherein theimage sensor is further configured to capture a third image when thelight source is being turned off; and a processing unit electricallycoupled to the image sensor and the light source, and configured tocalculate a first differential image between the first image and thethird image, and calculate a second differential image between thesecond image and the third image; calculate a differential image betweenthe first differential image and the second differential image, andidentify a darkest region in the differential image as an image of thelight source projected on the eyeball.
 2. The pupil tracking device asclaimed in claim 1, wherein the image sensor is configured to capturethe first image with a first exposure time and a first gain, and capturethe second image with a second exposure time and a second gain, whereinthe first exposure time is not equal to the second exposure time, or thefirst gain is not equal to the second gain.
 3. The pupil tracking deviceas claimed in claim 1, wherein the pupil tracking device is disposed ona head accessory.
 4. The pupil tracking device as claimed in claim 1,wherein the light source is configured to emit light in a firstbrightness value and a second brightness value alternatively at thedifferent times.
 5. The pupil tracking device as claimed in claim 1,wherein the processing unit is configured to identify the darkest regionin the differential image as the image of the light source projected onthe eyeball when a position of the darkest region in the differentialimage corresponds to a position of a brightest region associated withthe first reflection spot on the eyeball in the first differential imageor a brightest region associated with the second reflection spot on theeyeball in the second differential image.
 6. The pupil tracking deviceas claimed in claim 5, wherein the processing unit is further configuredto identify a pupil position according to the first differential imageor the second differential image.
 7. The pupil tracking device asclaimed in claim 6, wherein the processing unit is configured toidentify a plurality of pixels around a lowest gray level in the firstor second differential image and having gray levels within a gray levelrange as a pupil region, and calculate a gravity center or a center ofthe pupil region as the pupil position.
 8. The pupil tracking device asclaimed in claim 6, further comprising a memory unit configured to storepupil coordinate information associated with the image of the lightsource and the pupil position.
 9. A pupil tracking device, comprising: alight source configured to emit light toward an eyeball with differentbrightness values at different times to form a first reflection spot anda second reflection spot on the eyeball, respectively; an image sensorconfigured to capture a first image containing the first reflection spotand a second image containing the second reflection spot, wherein theimage sensor is further configured to capture a third image when thelight source is being turned off; and a processing unit electricallycoupled to the image sensor and the light source, and configured tocalculate a first differential image between the first image and thethird image, and calculate a second differential image between thesecond image and the third image; calculate a differential image bysubtracting the second differential image from the first differentialimage, and identify one of two darkest regions in the differential imagehaving a smaller area and corresponding to a position of a brightestregion in the first or second differential image as an image of thelight source projected on the eyeball.
 10. The pupil tracking device asclaimed in claim 9, wherein the pupil tracking device is disposed on ahead accessory.
 11. The pupil tracking device as claimed in claim 9,wherein the image sensor is configured to capture the first image with afirst exposure time and a first gain, and capture the second image witha second exposure time and a second gain, wherein the first exposuretime is not equal to the second exposure time, or the first gain is notequal to the second gain.
 12. The pupil tracking device as claimed inclaim 9, wherein the processing unit is further configured to identifythe other one of the two darkest regions in the differential imagehaving a larger area as a pupil image.
 13. The pupil tracking device asclaimed in claim 12, further comprising a memory unit configured tostore pupil coordinate information associated with the image of thelight source and the pupil image.
 14. The pupil tracking device asclaimed in claim 9, wherein the light source is configured to emit lightin a first brightness value and a second brightness value alternativelyat the different times.
 15. A pupil tracking device, comprising: a lightsource configured to emit light toward an eyeball with fixed brightnessvalue at different times to form a first reflection spot and a secondreflection spot on the eyeball, respectively; an image sensor configuredto capture, using a first exposure time, a first image containing thefirst reflection spot, and capture, using a second exposure timedifferent from the first exposure time, a second image containing thesecond reflection spot, wherein the image sensor is further configuredto capture a third image when the light source is being turned off; anda processing unit electrically coupled to the image sensor and the lightsource, and configured to calculate a first differential image betweenthe first image and the third image, and calculate a second differentialimage between the second image and the third image; calculate adifferential image between the first differential image and the seconddifferential image, and identify a darkest region in the differentialimage as an image of the light source projected on the eyeball.
 16. Thepupil tracking device as claimed in claim 15, wherein the image sensoris further configured to capture the first image using a first gain, andcapture the second image using a second gain different from the firstgain.
 17. The pupil tracking device as claimed in claim 15, wherein thepupil tracking device is disposed on a head accessory.
 18. The pupiltracking device as claimed in claim 15, wherein the processing unit isconfigured to identify the darkest region in the differential image asthe image of the light source projected on the eyeball when a positionof the darkest region in the differential image corresponds to aposition of a brightest region in the first differential image or abrightest region in the second differential image.
 19. The pupiltracking device as claimed in claim 18, wherein the processing unit isfurther configured to identify a pupil position according to the firstdifferential image or the second differential image.
 20. The pupiltracking device as claimed in claim 19, further comprising a memory unitconfigured to store pupil coordinate information associated with theimage of the light source and the pupil position.